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been  defined  by  the  Marine  Corps  after  a  careful  analysis  of  each  require¬ 
ment’s  structure*  The  categories  are: 

1.  Advance  preparation  and  education. 

2.  Combat  service  support  and  logistics  considerations. 

3.  Mission  planning  and  preparation. 

4.  Command  control  and  task  organization. 

5.  Execution. 

6.  Information  and  communication. 


Taking  the  category  model  as  a  starting  point,  there  is  still  the 
problem  of  estimating  K  probabilities  for  each  unit  (in  our  case  there 
are  75  units  in  the  study).  The  use  of  empirical  Bayes  methodology  has 
proven  useful  in  the  simultaneous  estimation  of  many  parameters  and  suggests 
itself  as  a  promising  tool  here.  We  will  demonstrate  that  an  appropriate 
Bayes  model,  described  below,  does  indeed  lead  to  improved  estimation  of 
the  category  probabilities. 


2 .  The  Category  Model 


For  the  N  =  75  units  in  the  present  study  we  define  X..  to  be  the 

ij 

number  of  satisfied  requirements  in  category  j  ,  for  the  i-th  unit.  Let 

M^^  be  the  number  of  requirements  in  category  j  that  the  i-th  unit  was 

tested  on.  Due  to  practical  constraints  units  are  generally  not  evaluated 
on  all  requirements.  If  0^^  is  defined  to  be  the  probability  of  unit  i 

satisfying  a  requirement  in  the  j-th  category  then  we  assume  that  has 

a  binomial  distribution,  further  assume  that  given 

9  E  (9  ,...,6  )'^  ,  X  ,...,X.„  are  independent.  These  two  assumptions 

IX  IK  IX  IK 

are  consistent  with  the  fact  that,  for  a  particular  unit,  performances  on 
different  requirements  are  independent.  Finally,  we  assume  that  X-,...,X_^ 

are  independent,  where  X.  E  (X. - , . . . ,X.^)  ,  i.e.,  the  performances  of 

r\jl  IX  IK 
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different  units  are  independent.  These  assumptions  are  summarized  as: 


ij 

ij  ij 

i=l 

il’  ■  ’ 

• ,  given 

0. 

are  independent  , 


are  independent. 


(1) 

(2) 

(3) 


The  most  critical  assumption  of  the  category  model  is  that  all  require 
ments  in  the  same  category  have  the  same  probability  of  being  satisfied. 
Zacks,  Marlow,  and  Barzily  (1981)  and  Zacks  and  Marlow  (1982)  have  concluded 
that  the  assumption  of  equal  probabilities  within  a  category  is  tenable  for 
a  ten  category  model  similar  to  the  one  that  we  propose.  We  use  a  cross- 
validation  technique  to  investigate  the  appropriateness  of  the  six  categories 
The  234  requirements  of  section  A  were  randomly  split  in  half  producing 

two  sets  of  count  vectors,  }  and  ,  ,X^^^  }  .  If  the 

probability  of  satisfying  a  requirement  is  a  function  only  of  its  category 

then  we  should  have  equal  probabilities  for  xf^^  and  xf^^  .  More  formally: 


i:  ij 

13 

(4) 

IJ  13 

(5) 

and  we  want  to  test  the  hypotheses, 

H  :  ; 

0  ij  » 

j— 1, • « • ,K 

(6) 

vs.  H  :  some  0^^^  =j=  0^?^ 

A  13  T  13 

(7) 
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3.  Verifying  the  Assumptions  of  the  Category  Model 

(1)  (2) 

For  a  particular  unit  and  category,  we  can  test  whether  6^^  =  0^^ 

by  using  Fisher’s  exact  test  (see  Fisher  (1925)).  This  test  is  illustrated 
in  Table  1  below. 

Table  1 

2x2  Contingency  Table  for  Testing  Whether 

e(i)^  q(2) 

ij  ij 


Satisfied 

Not  Satisfied 

First  Half 

x(i> 

IJ 

13 

-  xfi)) 

13 

IJ 

Second  Half 

X<2) 

13 

(M?2) 

13 

-  xf2)) 

13 

Mf2) 

IJ 

c. . 

IJ 

c. . 

IJ 

M.  . 
ij 

Fisher's  test  is  based  on  the  conditional  distribution  of  given 

ij 

C. .  ,  C. .  ,  ,  and  .  Under  H  ,  we  have 


ij 


ij 


r(l)  i — il 


IJ 


,  c. .  ,  c..  , 

13  '  13  13  13  13 


(8) 


and  the  conditional  significance  level  or  "p-value"  of  the  test  is  given  by 


a.  .  =  S  P{xf^^  =  x|c..  ,  C..  ,  , 

13  X  £  I  '  13  13  13  13 


(9) 
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where 


I  = 


=  jx:  P{X. 

'  1 


(1) 

ij 


=  X 


C.  .. 

ij' 


ij 


M 


<  P{xf^^  =  xf^^  |C 
iJ  ij  -  ij  13  '  ij 


C. 

ij 


M 


(1)  ^(2) 


13 


13 


}((10) 


After  performing  the  test  for  each  combination  of  (i,j)  we  are  left 
with  the  problem  of  combining  all  of  the  NK  =  450  p— values  to  make  an 
inference  about  defined  in  (6) •  We  use  two  different  approaches  to 

address  this  issue.  First  note  that  if  had  a  continuous  distribution 

then  we  could  use  the  well-known  result  (see  Fisher  (1925))  that 

-  2  £n  a.^  -v  x^[2]  ,  (11) 

2 

where  X  [v]  denotes  a  chi-square  random  variable  with  v  degrees  of 
freedom.  We  could  then  combine  the  tests  by  noting  that 


N  K 

Q  =  E  Z  (-2  £n  a. .)  (12) 

i=l  j=l 


2  (1) 

is  distributed  as  x  [21'TK]  .  Of  course  is  a  discrete  random  variable 

but  its  conditional  distribution  is  approximately  continuous  when 
^ii  ’  ^ii  ’  are  all  large. 


Another  approach  to  combining  the  p-values  is  to  note  that  under 


Hq  ,  each  is  independent  and  has  probability  .95  of  being  greater 

than  .05  (again  assuming  continuity).  Thus  if  we  define 


//of  a.,  greater  than  .05 

W  =  - iJ - 

NK 


then  W  is  distributed  as  B(NK,  .95)  if  0^^^  =  0^^^ 

13  ij 


(13) 


for  all  i,j  . 
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Table  2  summarizes  the  results  for  twenty  different  random  splittings 
of  the  234  requirements.  The  splittings  were  obtained  using  GGPER,  an  IMSL 

/s 

subroutine  which  generates  random  permutations,  and  the  were  computed 

using  IMSL  subroutine  CTPR.  All  computations  were  performed  on  an  IBM  370 
computer . 

If  we  first  consider  the  left  part  of  the  table,  which  includes  all 
cases  for  which  Fisher’s  test  is  meaningful,  we  see  that  all  values  of  W 
are  greater  than  .95,  and  all  values  of  Q  are  less  than  788  ,  the  number 
of  degrees  of  freedom  of  the  reference  chi-square  distribution.  These  results 
clearly  provide  no  cause  for  rejecting  the  hypothesis  of  equal  probabilities 
in  each  half.  Looking  at  the  right  part  of  the  table,  we  note  that  for  some 
splits  of  the  requirements,  W  is  less  than  .95.  Seven  of  the  twenty 
values  of  W  are  below  .95  but  none  are  more  than  1.6  standard  deviations 
below,  where 


SE(W)  =  /( .95)  ( .05) / (#of  applicable  tests)  .  (14) 

Using  the  Bonferroni  method  to  set  the  overall  significance  level  at  .10, 
none  of  the  twenty  W’s  provides  significant  evidence  against  the  hypothesis 
of  equal  probabilities  in  both  halves.  Note  also  that  all  twenty  of  the 
Q’s  are  less  than  the  corresponding  degrees  of  freedom  which  is  clearly 
consistent  with  the  null  hypothesis. 

In  summary,  we  find  that  the  assumption  of  equal  probabilities  for 
all  requirements  in  the  same  category  to  be  quite  tenable.  Using  the 
category  model  we  now  develop  an  empirical  Bayes  procedure  to  improve  the 
estimates  of  the  category  probabilities. 

4 .  Empirical  Bayes  Estimates  of  the  Category  Probabilities 

Our  chief  interest  is  in  estimating  0  ,...,6  ,  a  large  number  of 

multivariate  parameters.  Since  these  are  not  unrelated  parameters,  empirical 
Bayes  or  James-Stein  type  (see  James  and  Stein  (1961))  estimators  are 
candidates  for  reducing  the  overall  error  of  estimation  of  the  N  parameter 
vectors.  A  lucid  exposition  of  the  applicability  of  empirical  Bayes 
estimators  is  given  by  Morris  (1983) . 
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Table  2 

Combining  Fisher's  Tests  Across  All 
Evaluations  and  Categories 


Number  of  Number  of 


-  Cl')  (2) 

Cases  in  which  any  of  C  ,  C  ,  M:/  or  /  are  0  lead  to  a  degenerate 
distribution  for  .  These  cases  were  omitted. 


Only  cases  in  which 
are  included. 


C.  .  .  >  5, 

13  ij  13 


£  =  1,2  and 


C..  >  5 

13  13  13 


9 


^  =  1,2, 
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The  basic  assumptions  we  have  made  are  that  different  evaluations 

are  independent,  i.e.  X  , . . . ,X,  are  independent  random  variables,  and 

Oil  nA'i 

that  each  X..  ,  given  0.  ,  is  distributed  as  B(M  ,0  .)  .  If  the 
ij  i 

M. ,  are  all  moderately  large,  then  the  X..  are  approximately  normal 
i  J  ^  J 

and  many  well-known  results  can  be  applied.  To  further  simplify  the 
analysis  we  make  use  of  the  variance  stabilizing  transformation 

Y.  .  =  2  arcsin  (  V(X.-.  +  .5)/(M  +  1))  .  (15) 

ij  13  ^3 

Y  is  distributed  approximately  as  a  normal  variable  with  mean 

ij 


=  2  arcsin  (16) 

and  variance  (see  Johnson  and  Kotz  (1970)).  .  The  usefulness  of  this 

transformation  is  that  the  variance  of  Y^^  does  not  depend  on  any 

unknown  parameters.  In  this  transformed  scale  assumption  (2)  implies 
that  independent,  given  ?  •  •  • 

Y.  ==  (Y  ,...,Y  )  and  n.  =  (fj .  t  .  ,r|  ,„)  .  We  summarize  the  distribu- 

l-L  iJx  /\jl  11  iK 

tional  assumptions  by  stating  that 


Y  \r\  N(n.  ,D  ) 

r^l  i\,±  1 


i=l,. ..,N 


(17) 


where 


D.  =  diag  .  (18) 

KxK 


A  Bayesian  model  postulates  that  the  vectors  ri .  are  independently 

drawn  from  a  (prior)  distribution  which,  in  our  context,  can  be  thought 
of  as  representing  the  ’’superpopulation”  of  combat  units.  It  is 
convenient  to  assume  that  the  r] .  are  drawn  from  a  normal  distribution. 


-  8  - 
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i.e.,  H-i  >  •  •  •  >rixT  vectors 

'X.J-  f\P 


n.  ^  N  (y,Z)  ,  ,  (19) 

where  ]i  is  an  arbitrary  mean  vector  and  Z  is  an  arbitrary  positive 

definite  covariance  matrix*  It  should  be  noted  here  that  other  prior 
distributions  are  possible.  Leonard  (1972)  considers  models  where 
log  (O^j/ (l-0^j) )  is  normally  distributed;  Good  (1965)  allows  0^^  to 

have  a  beta  distribution.  While  these  alternative  models  are  somewhat 
tractable  in  a  single  category  situation,  it  is  difficult  to  incorporate 
a  dependence  structure  into  the  distribution  of  6.  in  either  case. 

Since  the  are  large  enough  for  the  assumption  of  normality  to  be 

reasonable,  there  are  no  compelling  reasons,  in  this  application,  for 
using  either  of  these  alternative  models. 

Using  (17)  and  (18),  the  predictive  or  unconditional  distribution 
of  Y.  is  N(y,  Z  +  D.)  and  the  posterior  distribution  of  n .  is 


|Y.  %  N(Y.  - 


B. 

1 


(Y  -  y  ) 

Oil 


(20) 


where 


Bi  E  D^(D^  +  Z)  ^  .  (21) 

If  y  and  Z  were  known,  the  Bayes  estimate  of  n*  >  assuming  squared 

Of  \P- 

error  loss,  would  be  the  posterior  mean, 


B.  -  ]i)  • 


(22) 


This  estimator  has  been  proposed  by  Efron  and  Morris  (1972)  for  the  case 
in  which  D^,,,.,D^  are  equal. 
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In  order  to  use  the  estimator  in  (22) ,  we  need  estimates  of 
y  and  E  .  The  unconditional  distribution  of  Y  leads  to  the  follow- 
ing  simple  estimates: 


y*  =  Y  .  (23) 

'Xj 


N  _  - 

E  (Y.  -  Y)  (Y.  -  Y)  -  D  , 


(24) 


where  Y=N^  E  Y.  ,D=N^  E  D.  .  Although  these  estimates  are  not 
i-i  1-1  ^ 

unreasonable,  they  are  not  the  most  efficient  ones  available  and  Z  has 
the  added  drawback  of  allowing  negative  estimates  of  variances.  We  instead 
suggest  estimates  of  y  ,  Z  derived  by  maximizing  the  predictive  likeli- 

'Xy 

hood  of  Y  ,  ...,Y  .  This  likelihood  function,  L‘(u,  Z;  Y.  ,.,,,Y„)  is 

0.1  '\P 

proportional  to: 


-1/? 

n  |I  +  D.  I  exp  {- 

i=i 


^  (Y.  -  y)"  (2  +  D.)  ^  (Y.  -  y)}  . 


(25) 


Instead  of  directly  maximizing  (25)  we  will  make  use  of  the  Ell  algorithm 
of  Dempster,  Laird,  and  Rubin  (1977).  The  Ell  algorithm  is  designed  for 
missing  data  situations  but  it  can  be  applied  here  if  we  think  of 

as  the  missing  "data."  The  algorithm  consists  of  an  E-step, 

computing  the  expectations  of  the  sufficient  statistics  (n.  ,  •  .  .  ,r],J 

ol 

given  the  observed  data  together  with  the  current  estimates  of  the 

parameters  (y,Z)  ,  and  an  M-step  in  which  the  likelihood  of  the  complete 
o 

data  based  on  the  estimated  sufficient  statistics  is  maximized.  To  define 

the  algorithm  more  specifically,  let  ,  Z^^^  be  the  current  estimates 

a. 

of  (y,  Z)  after  p  iterations.  From  (22)  and  the  independence  of 
ri-,  , .  . .  ,r].T  9  the  E-step  is  given  by 


10  - 
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.  E 


(n.  |y 

% 


(p)  ?(p) 


Y,,...,Y  )  =  Y,  -  B; 


(P) 


(Y  - 


(26) 


From  standard  results,  if  were  in  fact  the  observed 

0/1 

values  of  r)^  ,  .  .  •  jD^t  »  the  maximum  likelihood  estimates  of  y  ,  E 

'\/-L  f\j 

would  be 


^(p+i) 

'\j 


N 

E 

i=l 


^  (p+1) 


(27) 


J(P+1)  .  ^-1 


N 

E 

i=l 


(n^ 


(p+i) 


r\j± 


O/ 


(28) 


Equations  (26)  -  (28)  define  the  algorithm  for  our  problem.  Using  y*  ,  E*  , 

defined  by  (23) ,  (24)  as  initial  estimates  led  to  convergence  of  the 

algorithm  (to  four  decimal  places  in  all  components  of  y  and  E)  in 

o- 

nine  iterations.  Table  3  gives  the  initial  and  final  estimates-  Note 
that  y  and  y*  are  very  close  but  there  are  substantial  differences 

between  E  and  E*  . 


Using  (22)  we  obtain  the  empirical  Bayes  estimates 


n.  =  Y  -  B  (Y  -  y)  ,  , 

0/1  1  oi  o 


(29) 


where  B.  =  D.  (D.  +  E) 
1  11 


-1 


Our  interest  is  in  estimating  0,  and  the 

O/i 


natural  estimates  to  take  are 


).  .  =  [sin  (n.  J2)y 
ij  ij 


i=l, . . • ,N  ;  j=l, . . . ,K 


(30) 


although  these  are  not  exactly  the  Bayes  estimates  of  the 

assumed  model. 


11  - 
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Table  3 


Initial  and  Final  Estimates  of  Vi,  2 


I*' 


Z  : 


2.3916 

2.3837 

2.2332 

2.4693 

2.2211 

2.4081 

2.3943 

2.3843 

2.2338 

2.4691 

2.2233 

2.4097 

.0693 

.0426 

.0648 

.0558 

.0597 

.0428 

.0087 

.0474 

.0475 

.0616 

.0376 

.0856 

.0809 

.0574 

.0449 

.0866 

.0589 

.0509 

.1182 

.0553 

- 

.0471 

To461 

.0370 

.0557 

10573 

:0541 

.0399 

.0297 

.0447 

.0460 

.0434 

.0321 

.0673 

.0692 

.0653 

.0482 

.0712 

.0672 

.0496 

.0634 


.0468 

.0346 
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5%  Cross-Validation  of  the  Bayesian  Model 

The  motivation  for  using  empirical  Bayes  estimates  instead  of 
estimating  the  probabilities  with  the  relative  frequencies  is  the  hope 
that  the  overall  error  across  all  units  will  be  smaller •  Other  empirical 
studies  (see  Efron  and  Morris  (1975)  and  Fay  and  Herriot  (1979))  have 
demonstrated  that  empirical  Bayes  techniques  do  lead  to  improved 
estimation.  In  both  of  these  studies  the  true  parameter  values  were 
available  so  the  actual  error  of  the  estimates  could  be  computed.  We  do 
not  have  this  luxury  so  we  will  again  use  cross-validation  to  compare  the 
Bayes  estimates  with  the  sample  proportions. 

As  described  in  section  2,  we  split  the  234  requirements  into  two 
halves.  The  first  half  was  used  to  estimate  the  probabilities  and  the 
relative  frequencies  in  the  second  half  were  compared  with  these  estimates. 
/sB 

Define  p,,  to  be  the  Bayes  estimate  computed  from  the  first  half  and 


let  p..  denote  the  (modified)  sample  proportion  from  the  first  half, 


i.e. 


(31) 


Let  p..  denote  the  (modified)  sample  proportion  from  the  second  half. 


(32) 


For  each  unit  we  would  like  measures  of  agreement  between  n .  and  n 


and  between  p.  and  p*  .  We  propose  two  measures: 


(33) 
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In  a  similar  manner  define  C.  and  L.  by  replacing  p®  with  p 

11  ij  ^ij 

in  (33)  and  (34) .  Note  that  is  the  Pearson  chi-square  statistic 

and  L.  the  likelihood-ratio  statistic  for  comparing  p.  and  p. 

By  using  these  measures  we  are  attempting  to  incorporate  the  variability 
of  p  into  the  comparisons.  Bishop,  Fienberg,  and  Holland  (1975)  give 

a  complete  discussion  of  the  merits  of  using  these  statistics  to  compare 
observed  and  estimated  probabilities. 

Table  4  summarizes  our  findings.  We  have  summed  C  ,  C  L 

1  i  i 

g 

and  over  the  N=75  evaluations.  The  results  are  strikingly  in  favor 

of  the  Bayes  estimates.  For  every  one  of  the  20  splits  of  the  require¬ 
ments  the  Bayes  estimates  produced  a  smaller  total  "error"  as  measured  by 
either  C  or  L  .  We  have,  for  a  number  of  splits,  looked  at  the  75 
values  of  C  and  L  for  each  unit  and  it  is  clear  that  the  Bayes 
estimators  are  providing  protection  against  gross  errors  in  estimating 
Pij  9  the  large  values  of  or  are  almost  exclusively  observed 

for  p. .  as  opposed  to  p..  - 

In  summary,  we  have  found  that  the  empirical  Bayes  methodology 
leads  to  a  very  tractable  means  of  estimating  a  collection  of  vector 
parameters  and  offers  a  considerable  improvement  over  the  typical  method 
of  estimating  each  vector  separately.  To  our  knowledge  this  is  the  first 
empirical  study  that  assesses  the  efficacy  of  empirical  Bayes  estimators 
for  vector  parameters.  We  hope  that  it  leads  to  a  greater  usage  of  these 
techniques  in  the  future. 
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Table  4 

Cross-Validation  of  Bayes  and  Ordinary 
Estimates  of  Probabilities 


Split 

75 

E  c. 

i=l  ^ 

Z  C® 

1=1  ^ 

75 

Z  L. 
i=l  ^ 

R 

Z  L® 
i=l  ^ 

1 

824.3 

539.2 

614.6 

506.7 

2 

898.5 

547.2 

702.7 

512.8 

3 

788.1 

563.7 

678.0 

555.1 

4 

784.9 

572.6 

596.5 

513.8 

5 

809.2 

533.9 

709.9 

552.8 

6 

780.4 

439.6 

667.2 

470.4 

7 

906.3 

614.9 

760.4 

587.6 

8 

818.3 

620.3 

686.1 

594.9 

9 

815.0 

582.8 

704.4 

565.8 

10 

895.7 

665.4 

655.0 

558.3 

11 

863.0 

587.3 

657.9 

533.4 

12 

970.2 

588.8 

637.2 

511.0 

13 

850.5 

563.5 

673.8 

508.6 

14 

704.8 

541.1 

656.4 

565.1 

15 

1213.8 

735.9 

814.7 

636.8 

16 

989.4 

695.7 

702.6 

620.0 

17 

880.6 

554.8 

606.9 

495.5 

18 

956.5 

586.6 

753.2 

564.2 

19 

1125.1 

611.8 

793.9 

570.8 

20 

914.7 

574.1 

652.2 

529.1 
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